System Message Free until April 26th, 2025

Setting Up AI Cluster

Create a Lossless Fabric for the AI workloads

After installing Cisco NDFC, you decide to test NDFC automation capabilities by deploying BGP EVPN fabric if the company decides to expand AI workloads across several sites in the future.

In this task, you will deploy the EBGP fabric with configured AI/ML QoS and queuing policies. This way, you will achieve lossless behavior in the network, which is required for the AI workload communication over RoCEv2. You will use Cisco NDFC templates to automate the deployment of fabric and QoS configuration.

Step 1

On your desktop, locate and open the Chrome web browser. Then, navigate to https://10.0.0.70.

Answer

You should see the login page for the Cisco Nexus Dashboard.

Accept any alerts regarding privacy by first clicking Advanced and then clicking Proceed to 10.0.0.70 (unsafe), as shown in the figures.

Step 2

Enter admin and C!sco123 in the username and password fields, respectively. Click the Login button.

Answer

You will see the Welcome, admin page with the Journey tab selected. The Getting Started Map is partially complete and it shows the next step, which is adding a fabric.

If you see an alert regarding privacy concerns, click Advanced, followed by clicking Proceed to 10.0.0.70 (unsafe), as depicted in the figures.

Step 3

Click Admin Console in the top bar and choose Fabric Controller from the menu.

Answer

You will see the Overview page of the Cisco Nexus Dashboard Fabric Controller.

Step 4

To manage the NDFC fabrics, hover over the Manage section in the left navigation panel and choose Fabrics.

Answer

Notice that the widgets show no information because you have not created fabrics or added switches to the Cisco Nexus Dashboard.

Step 5

Click the Actions button, and choose Create Fabric from the drop-down list.

Answer

Step 6

Name the fabric AI-Fabric, and then click the Choose Fabric button.

Answer

Step 7

Click the BGP Fabric template to highlight it, and then click the Select button.

Answer

The NDFC templates provide an already populated configuration to easily deploy complex fabrics across your network devices. In this task, you will use the BGP Fabric template to deploy the EBGP underlay with QoS policies to ensure lossless traffic.

Step 8

In the General Parameters tab, enter 65000 in the BGP ASN for Spines field. Verify that Multi-AS is chosen from the BGP AS Mode drop-down list, and check the Allow Same ASN On Leafs option.

Answer

Even though the Multi-AS setting allows you to use a unique ASN for each leaf, you also checked the option Allow Same ASN On Leafs. This option lets you configure the same BGP ASN on leafs. The same ASN on leaf switches is typically used when you want to create a single virtual port channel (vPC) pair.

Step 9

Click the EVPN tab to inspect the configuration. Make sure that the Enable EVPN VXLAN Overlay, Enable VXLAN OAM, and Enable Tenant DHCP options are enabled.

Answer

You should see that the EVPN overlay settings are already enabled by the BGP Fabric template.

Step 10

Click the Protocols tab, and check the Enable BFD and Enable BFD for BGP check boxes.

Answer

The Bidirectional Forwarding Detection (BFD) option enables faster convergence times when the fabric is being created.

Step 11

Click the Advanced tab. Scroll down and choose Enable from the Greenfield Cleanup Option drop-down list.

Answer

The Greenfield Cleanup Option allows you to clean the switch configuration when adding it to the NDFC without a switch reload. This option is recommended only for environments with virtual switches such as the Cisco Nexus 9000v that you use in this task.

Step 12

Scroll further down and check the Enable AI/ML QoS and Queuing Policies check box. Choose AI_Fabric_QOS_25G from the AI/ML QoS & Queuing Policy drop-down list.

Answer

The AI_Fabric_QOS_25G template provides the QoS configuration that will be applied to the entire fabric, including PFC and ECN configuration. The PFC and ECN congestion mechanisms need to be enabled in tandem to ensure lossless communication across the fabric. You will inspect the QoS configuration in closer detail when you deploy the configuration to the switches.

Note

In this scenario, you selected the 25G policy, but typically you would select the template that reflects the speed of the physical ports on your switches.

Step 13

Open the terminal by pressing Ctrl+Alt+T or by clicking the Terminal icon in the bottom taskbar on the jump host desktop.

Answer

You will see the terminal window.

Step 14

Type cat freeform.conf in the terminal and press the Enter key.

Answer

Your terminal will now display the contents of the freeform.conf file.

Step 15

Select the lines with the content, right-click, and choose Copy from the menu to copy the contents to the clipboard.

Answer

Step 16

Back in your browser, scroll to the bottom, and paste the freeform configuration you have in the clipboard to the Intra-fabric Links Additional Config.

Answer

The freeform configuration window allows you to specify the commands that will be applied to intra-fabric links in addition to the configuration provided by the NDFC templates. These commands will enable PFC and QoS traffic classification on the interfaces between spine and leaf switches. The priority-flow-control watch-dog-interval on command enables the watch-dog timer that will monitor PFC frames and prevent potential PFC storms in the fabric.

Step 17

Click the Save button in the lower-right corner of the page to save the fabric configuration.

Answer

You will land back on the Fabrics page that will now display the AI-Fabric that you have just configured.

Add Switches to the Fabric

Impressed with how easy it is to configure the BGP fabric with Cisco NDFC fabric templates, you decide to add your Cisco Nexus switches to the BGP fabric. But before pushing the configuration to the switches, you want to verify what kind of QoS configuration is NDFC deploying.

Step 18

In the bottom toolbar, locate and click the terminal emulator icon.

Answer

You will begin by connecting to the switch and checking the QoS configuration.

Step 19

Use the ssh admin@172.16.0.10 command to connect to the leaf01 switch. If prompted to continue connecting, enter yes and use C!sco123 as the password.

Answer

You should get an output with the licensing information and a prompt that is set to the leaf01 switch. You will see that Cisco Nexus 9000v virtual switches are used in this task.

Step 20

Use the show run ipqos command to check if the switch has the QoS configuration applied to it.

Answer

You should see empty output. You can verify other switches using the same method. At this point, all switches should have no QoS configuration applied to it. In the following steps, you will use Cisco NDFC to apply the configuration by simply adding switches to the AI fabric that you created in the previous task.

Step 21

Navigate back to the browser where you have Cisco NDFC opened. Double-click the AI-Fabric to open Fabric Overview.

Answer

You should see that the fabric is healthy, but has no switches added to it.

Step 22

In the top-right corner, click the Actions button, and choose Add Switches from the drop-down list.

Answer

Step 23

Enter the IP range 172.16.0.10-20 in the Seed IP field, admin as the username, and C!sco123 as the password. Verify that the Preserve Config check box is unchecked.

Answer

The provided IP range should discover both leafs and spine. You can always access their information in the lab job aid at the beginning of the lab guide. Cisco NDFC will automatically discover all devices within the seed range and devices that are within the defined Max Hops range from the seed switches. The Preserve Config check box allows you to add the switch to the fabric without losing any configuration. When the box is not checked, the switch configuration is cleared during the import process, which removes all configuration except the management details and reboots the switch.

Step 24

Click the Discover Switches button in the bottom-right corner.

Answer

Step 25

Click Confirm in the warning alert to proceed.

Answer

After a few seconds, you should see the discovered devices in the Discovery Results table. Specifically, you should see leaf01, leaf02 and spine01 discovered and their status reported as Manageable. Notice that NDFC pings all IP addresses within the provided range and reports the details about the device such as Serial Number and Model.

Step 26

Click the check box next to the Switch Name to choose the leaf01, leaf02, and spine01 switches. Then click the Add Switches button in the lower-right corner.

Answer

In the process of importing switches, Cisco NDFC erases the configuration on the switches and reboots it. The import process can take up to 10 minutes with physical switches, but with Cisco Nexus 9300v virtual switches the process will be finished within a few minutes.

Once the import process is complete, the switch Status changes to Switch Added, and the Progress bar changes to Green.

Step 27

Close the window by clicking X in the upper-right corner.

Answer

You should land back on the Fabric Overview window.

Step 28

Navigate to the Switches tab, and verify that all three switches are now added to the fabric.

Answer

If the Discovery Status column reports Unreachable, choose the unreachable switches, click the Actions button, select Discovery, and click the Rediscover button. By doing rediscovery, Cisco NDFC attempts to reconnect to the switch and obtain switch details. The Discovery Status on all three switches should be OK.

Note

The Operational Status column might indicate Minor alert status because some interfaces are initialized as down. This status has no effect on the lab task.

Step 29

Choose the spine01 switch, click the Actions button, and choose Set Role from the drop-down list.

Answer

By default, NDFC will assign leaf role to all newly added switches, including the spine switches. Spine or any other device roles are assigned manually by selecting them and changing the role. You can select one or multiple devices at once.

Step 30

In the Select Role pop-up window, click Spine to highlight the choice. Then click the Select button in the lower-right corner of the pop-up window.

Answer

After selecting the role, you should see the Warning window advising you to trigger the Recalculate and Deploy action.

Click OK to close it. The spine01 switch is now properly assigned with the spine role.

Step 31

Click the upper, dark blue Actions button that is located beside the Refresh icon, and choose Recalculate and Deploy from the drop-down list.

Answer

The Recalculate and Deploy action will start calculating the configuration that will be applied to the devices.

After the window with calculation progress closes, you should see the Deploy Configuration window.

The Fabric Status column will report the Out-of-Sync status, indicating you have pending configuration that can be deployed to the added devices. The Pending Config column shows how many lines of the configuration are pending to be deployed.

Note

The order of the devices that you see in the table might be different compared to the lab guide.

Step 32

Click the blue link under the Pending Config column in the leaf01 row to check which commands will be applied to leaf01.

Answer

After scrolling down, you should see the QoS configuration that will be applied to leaf01. This configuration is provided by the AI/ML QoS template that you included in the BGP Fabric template.

In the configuration below, the class-map type qos command ensures that all traffic marked with DSCP 26 will be classified as RoCEv2 traffic. Also, all traffic marked with a DSCP value of 48 will be marked as CNP (Congestion Notification Packet). The goal is to classify the RoCEv2 and CNP traffic, so that switches can put it in the right queue.

class-map type qos match-any ROCEv2
  match dscp 26
class-map type qos match-any CNP
  match dscp 48

The policy-map configuration classifies appropriate traffic to the right queue. In this example, RoCEv2 traffic is classified to queue 3, and CNP control traffic is classified to the strict-priority queue 7 to deliver congestion notification packets as quickly as possible.

policy-map type qos QOS_CLASSIFICATION
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0

After traffic classification, you should see the queuing and scheduling configuration. The queue 3 (c-out-8q-q3) is assigned 50% of the bandwidth. The minimum threshold is set to 100 KB, the maximum threshold to 700 KB, and if there is congestion happening, mark the packets with ECN. The queue 7 (c-out-8q-q7) has priority 1 assigned to it. This queue handles the CNP packets.

policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q6
    bandwidth remaining percent 0
  class type queuing c-out-8q-q5
    bandwidth remaining percent 0
  class type queuing c-out-8q-q4
    bandwidth remaining percent 0
  class type queuing c-out-8q-q3
    bandwidth remaining percent 50
    random-detect minimum-threshold 100 kbytes maximum-threshold 700 kbytes drop-probability 40 weight 0 ecn
  class type queuing c-out-8q-q2
    bandwidth remaining percent 0
  class type queuing c-out-8q-q1
    bandwidth remaining percent 0
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 50
  class type queuing c-out-8q-q7
    priority level 1

To enable the PFC pause frames for a specific queue, an additional QoS configuration should be applied. In this example, the PFC pause frames are enabled on queue 3 and will be generated if congestion happens.

policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    pause pfc-cos 3
    mtu 4200

The service-policy commands are required to apply queuing and QoS configurations systemwide. So, if there is congestion, packets will be marked with ECN. The ports with PFC will receive those frames and send pause frames if congestion thresholds are triggered.

system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

Scroll further down to verify the configuration of interface Ethernet1/1. To implement end-to-end lossless fabric, PFC must be enabled on all interfaces across the network. The RoCEv2 traffic classification is enabled at the interface level, including the logical Network Virtual Interface (nve1) where traffic encapsulation and de-encapsulation occurs.

interface ethernet1/1
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
  service-policy type qos input QOS_CLASSIFICATION
  no switchport
  ip address 10.4.0.2/30
  description connected-to-spine01-Ethernet1/1
  mtu 9216
  ip pim sparse-mode
  no shutdown
interface nve1
  source-interface loopback1
  host-reachability protocol bgp
  service-policy type qos input QOS_CLASSIFICATION

Step 33

Click the Close button in the bottom-right corner to close the Pending Config window.

Answer

Step 34

Click the Deploy All button in the bottom-right corner of the Deploy Configuration - AI-Fabric window.

Answer

Configuration deployment will take a couple of minutes. The Progress column will provide you with progress information. Once the changes are deployed, you should see the Success status in the Progress column.

Step 35

Click the Close button to return to the Fabric Overview window.

Answer

Newly added switches should now have the In-Sync Config Status.

Step 36

Select the leaf01 switch, click the Actions button to open the drop-down list, hover over More, and click Show Commands.

Answer

Step 37

From the Commands drop-down list, choose the show command.

Answer

Note

Once switches are added to Cisco NDFC, you can still use the switch CLI to run commands. In this example, Cisco NDFC is used to run commands on the switch.

Step 38

Under Show Command 1, enter the run ipqos command, and click Execute in the bottom-right corner to verify if the QoS configuration was applied to leaf01.

Answer

The Switch show commands window allows you to run various show commands and verify the switch running configuration. Notice that the QoS configuration is now applied to the switch.

Step 39

Click the X button in the top-right corner to close the window.

Answer

Configure the Overlay Network

After deploying the BGP fabric on the switches and inspecting the AI/ML QoS policy configuration, you are gaining confidence in the automation capabilities of Cisco NDFC. You decide to also configure the EBGP overlay network to distribute reachability information about the endpoints. You will continue to use Cisco NDFC policies to configure leaf BGP ASNs and add your spine switches as EBGP neighbors to your leaf switches and the other way round.

Step 40

In the Fabric Overview window, click the Policies tab.

Answer

You should see the list of all policies that are applied to the fabric. The policies provide an easy way to deploy or change configuration across the fabric.

Step 41

Click the white Actions button, and choose Add Policy from the drop-down list.

Answer

Step 42

Choose the leaf01 and leaf02 switch by clicking the check boxes next to their names, and click Next in the bottom-right corner.

Answer

Make sure you choose only leaf switches, because you will deploy the policy that will configure BGP ASNs on the leafs.

Step 43

In the Create Policy window, click the No Policy Selected button.

Answer

The Policy Template window should appear with the list of all defined policy templates in Cisco NDFC.

Step 44

Type asn in the search bar to find the leaf_bgp_asn policy. Click the policy to highlight it and click Select.

Answer

You should see the Create Policy window after selecting the policy.

Notice that now the Create Policy window shows the Leaf BGP AS # field.

Step 45

In the Leaf BGP AS # field, enter 65010, leave all other parameters at their default values, and click the Save button in the bottom-right corner.

Answer

After a few seconds, you should be redirected to the Policies tab and see a Success pop-up window in the bottom-right corner.

Step 46

Under the Policies tab, click the Filter by attributes bar to open a drop-down filter list, and select Template.

Answer

After selecting Template, the filter will immediately provide another drop-down list.

Note

The filter field provides suggestions when typing and supports the autocomplete command when pressing the Tab key on the keyboard.

Step 47

Choose contains from the drop-down list, and then type asn to complete the filter. Press Enter to search.

Answer

The filter field provides suggestions when typing and supports the autocomplete command when pressing the Tab key on the keyboard.

After performing the search, among the filtered policies, you will find the leaf_bgp_asn and bgp_asn policies that are created for each leaf switch. Notice that even though you configured only the leaf_bgp_asn policy, an additional bgp_asn policy was created for both leafs.

To configure the policies for EBGP neighboring between the leafs and spine, first you need to find out the IP addresses of the Loopback0 interfaces. The NDFC BGP template configured the loopback interfaces when you added switches to the fabric. You need to collect the IPs.

Step 48

Navigate to the Interfaces tab in the Fabric Overview window, click the Filter by attributes bar, and choose Interface.

Answer

Step 49

Click the == filter from the drop-down list, type Loopback0 and press Enter to search for the interfaces.

Answer

Step 50

Move the horizontal scroll bar to the right to see the IP/Prefix column and collect the IPs.

Answer

You should get similar outputs as shown below. In this example, the leaf01 Lo0 interface has IP address 10.2.0.1, the leaf02 Lo0 interface has IP address 10.2.0.3, and the spine01 Lo0 interface has IP address 10.2.0.2.

Note

Cisco NDFC assigns the Lo0 IP addresses when you add switches to the fabric, and your IPs might be different than shown in this example.

Step 51

In the Fabrics Overview page, choose the Policies tab. Click the white Actions button and choose Add Policy from the drop-down list.

Answer

Step 52

Choose the leaf01 and leaf02 switches and click Next in the bottom-right corner.

Answer

Step 53

Click the No Policy Selected button to select the policy template.

Answer

Step 54

In the search bar, type ebgp_overlay, click the ebpg_overlay_leaf_all_neighbor policy to highlight it, and click the Select button.

Answer

You should be back at the Create Policy page. Notice that you have two new fields to populate.

Step 55

In the Spine/Super Spine IP List field, enter the IP address that you have collected from the Lo0 interface on the spine01 switch. The BGP Update Source Interface by default should have the Loopback0 interface selected. Click Save in the bottom-right corner.

Answer

In this example, the IP is 10.2.0.2, but in your environment spine01 might have different IPs. Make sure to enter the IP that you have previously collected from the Interfaces tab.

After a couple of seconds, you should land back to the Policies tab and see a notification that the policies were successfully added.

Now you need to create a similar policy for the spine01 switch.

Step 56

From the Policies tab, click the white Actions button and choose Add Policy from the drop-down list.

Answer

Step 57

Check the check box next to the spine01 switch and click the Next button.

Answer

Step 58

Click the No Policy Selected button to select the policy template.

Answer

The Select Policy Template window should appear.

Step 59

In the search bar, type ebgp_overlay, click the ebpg_overlay_spine_all_neighbor policy to highlight it, and click the Select button.

Answer

In the Create Policy page, you should now see three additional fields to populate.

Step 60

In the Leaf IP List field, enter the IP addresses that you have collected from the Lo0 interfaces on the leaf01 and leaf02 switch. In the Leaf BGP ASN field, enter both leaf ASN numbers in a comma-separated list: 65010, 65010. The BGP Update Source Interface by default should have the Loopback0 interface selected. Click Save in the bottom-right corner.

Answer

In this example, leaf Lo0 IPs are 10.2.0.1 and 10.2.0.3, but in your environment, interfaces might have different IPs. Make sure to enter the Lo0 IPs that you have collected from the Interfaces tab.

After a couple of seconds, you should land back to the Policies tab and see a notification that the policies were successfully added.

After you configured the policies, you will need to deploy these changes to the switches. For any kind of configuration changes you make on fabric or policy level, Cisco NDFC must calculate the appropriate set of commands to reflect those changes.

Step 61

Click the blue Actions button in the top-right corner, and choose Recalculate and Deploy from the drop-down list.

Answer

You should see the recalculation progress in the temporary pop-up window.

After the calculation is done, you should be forwarded to the Deploy Configuration window. Because there are pending configuration changes on leaf01, leaf02 and spine01, Cisco NDFC is reporting the Fabric Status as Out-of-Sync on all three devices. If you want to inspect the pending BGP configuration, you can click the blue link under the Pending Config column for each device.

Step 62

Click Deploy All in bottom-right corner to deploy the configuration.

Answer

You should see the progress of the configuration deployment.

It will take Cisco NDFC a few seconds to deploy the configuration. When the configuration is deployed, you should see the Deployment completed status.

Step 63

Click the X button in the top-right corner to close the window.

Answer

You will return to the Fabric Overview page with the Policies tab selected. The spines and leafs now have a functioning EBGP overlay configured.

Configure the Network for AI Workloads

Having EBGP overlay configured and operational, you start thinking how to connect the AI cluster to the fabric. Because the AI cluster will generate a lot of traffic and process many sensitive documents, you decide to create a dedicated VRF and AI network subnet. This way, you will segregate the AI cluster from other workloads. You will also attach the AI network to the switch interfaces, so that endpoints Student VM and PC-01 get access to the network. Finally, by testing communication from Student VM (attached to Eth1/3 on leaf01) to PC-01 (attached to Eth1/3 on leaf02), you will test the end-to-end connectivity to confirm if the network is configured properly.

Note

The lab environment is using Cisco Nexus 9000v virtual switches. These switches allow you to configure QoS and congestion mechanisms for the AI network just like you would on the physical Cisco Nexus 9000 switches. Nonetheless, virtual Cisco Nexus 9000v switches cannot implement QoS policies and apply congestion mechanisms when congestion happens, that is only possible on physical Cisco Nexus switches. Therefore, within this lab exercise you will test the network connectivity for the configured network.

Step 64

In the Fabric Overview window, click the VRFs tab. Click the white Actions button and choose Create from the drop-down list.

Answer

You should see a new Create VRF window.

Notice that Cisco NDFC will automatically select the default VRF template and provide some default values for the VRF configuration.

Step 65

Change the VRF Name to AI_VRF_50000. Click the Propose VLAN button to allow Cisco NDFC to suggest the VLAN ID from the pool, and leave all other fields with the default values. Click Create in the bottom-right corner.

Answer

Because you are deploying this VRF for AI workloads, this task suggests the AI_VRF_50000 name, but you can choose any VRF name that you want.

After clicking Create, you will be redirected back to the VRFs tab, which is now showing the VRF that you have created. You should see the VRF status as N/A because it is not yet deployed on the switches.

Step 66

Click the Networks tab, then click the white Actions button and choose Create from the drop-down list.

Answer

You should see a new Create Network window. Notice that Cisco NDFC already selected the default Network template with the default values populated in the provided fields to speed up the configuration.

Step 67

Change the Network Name to AI_Network_30000 and click the Propose VLAN ID button to get the available VLAN ID from the pool. Type 192.168.1.254/24 in the IPv4 Gateway/NetMask field and AI_Workloads in the VLAN Name field. To confirm network creation, click the Create button.

Answer

Because you are creating a dedicated network and VLAN ID for the AI workloads, this example suggests using the AI_ prefix for the network and VLAN name. You can choose other names, but it is good practice to define descriptive names that allow you to easily identify their purpose. Also notice the IP of the gateway that you defined—it is the default gateway your AI workloads will use to access the network.

After clicking Create, you will be redirected back to the Networks tab.

You should see your network for AI workloads in the Networks tab. The Network Status is reported as N/A because the network is not yet assigned to the interfaces and deployed.

Step 68

At the bottom of the screen, locate and click the terminal emulator icon, and type the ip a command to list the network interfaces of Student VM.

Answer

The ip a command will list all interfaces on the Student VM. Notice four interfaces. One of them is the ens224 interface, which is configured with IP address 192.168.1.1/24. This IP address is from the same subnet as your AI network.

Step 69

Type the ip r command to list the routing table of Student VM.

Answer

Notice that the default gateway IP address 192.168.1.254 is already configured. This gateway IP address is the same that you configured for your AI network.

Step 70

Type ping 192.168.1.254, press Enter to check the reachability of the default gateway, and after 10 seconds press Ctrl+C to stop the pings. You will go back to the Cisco NDFC and leave the terminal window open.

Answer

You should see the Destination Host Unreachable message for each ping and 100% packet loss because the network is not yet deployed on leaf interfaces.

Step 71

At the bottom of the screen, locate and click the browser icon to return to your Cisco NDFC tab.

Answer

You should land back on the Fabric Overview Networks tab.

Step 72

In the Fabric Overview window, click the Interfaces tab to see all physical and virtual interfaces on all the switches in the fabric that Cisco NDFC discovered.

Answer

You should see the various attributes of the interfaces. Without applying a filter, it might be difficult to find specific information or interface because this view lists all interfaces in the fabric.

Step 73

Click Filter by attributes. Create the Interface == Ethernet1/3 filter and press Enter to filter interfaces.

Answer

You should see three interfaces after applying the filter. The interfaces will be Down because they are not yet configured or attached to the network.

In the following steps, you will configure the Ethernet1/3 interfaces on leaf01 and leaf02 as access interfaces. You will attach them to the AI network, so that Student VM (connected to Ethernet1/3 on leaf01) and PC-01 (connected to Ethernet1/3 on leaf02) can communicate over the 192.168.1.0/24 AI network.

Step 74

Select the Ethernet1/3 interfaces on the leaf01 and leaf02 switches, click the white Actions button, and choose Edit from the drop-down list.

Answer

Step 75

Click the int_trunk_host string under the Policy field to change the policy template.

Answer

You should see the Select Attached Policy Template window.

By default, Cisco NDFC is using the int_trunk_host policy for this interface. You will reconfigure this interface as access interface by choosing a different interface template.

Note

Notice that the Edit Interfaces window shows you are editing 1 of 2 Selected Interfaces, specifically you are editing the leaf01 Ethernet1/3 interface. After you are done with editing this interface, you will repeat the process for the leaf02 Ethernet1/3 interface.

Step 76

Type access in the search bar, click the int_access_host policy template to select it, and click the Select button when you are done.

Answer

After clicking the Select button, you should land back on the Edit Interfaces window. Notice the different configuration options available after you have changed interface policy from trunk to access type.

Step 77

In the Access Vlan field, enter 2300. In the Interface Description field, type Link to Student VM or something equally descriptive.

Answer

Make sure that the Access Vlan value is the same ID value as you specified for your AI network VLAN ID.

Step 78

Scroll further down to make sure that the Enable Interface, Enable Priority Flow Control, and Enable QoS Configuration check boxes are checked. Click the Save & Next button in the bottom-right corner.

Answer

The PFC and QoS configuration needs to be applied end-to-end to have lossless fabric. This includes configuring QoS and PFC on access interfaces as shown in this example. You have already configured the spine-to-leaf connections in the previous tasks when you configured the BGP fabric template.

After clicking the Save & Next button, you will repeat the same configuration steps, but this time for the Ethernet1/3 interface of the leaf02 switch.

Step 79

Click the int_trunk_host string under the Policy field to change the policy template.

Answer

Notice that you are now editing the Ethernet1/3 interface of the leaf02 switch. By default, Cisco NDFC is using the int_trunk_host policy for this interface. You will reconfigure this interface as access interface by selecting a different interface template.

Step 80

Type access in the search bar, click the int_access_host policy template to select it, and click the Select button when you are done.

Answer

Step 81

In the Access Vlan field, enter 2300. In the Interface Description field, type Link to PC-01 or something equally descriptive.

Answer

The configuration process for the Ethernet1/3 interface of the leaf02 switch will be very similar. Make sure that the Access Vlan value is the same ID value as you specified for your AI network VLAN ID and the leaf01 Ethernet1/3 interface. The description is now Link to PC-01, because it is the interface where PC-01 is connected.

Step 82

Scroll further down to make sure that the Enable Interface, Enable Priority Flow Control, and Enable QoS Configuration check boxes are checked. In the bottom-right corner, click the Save button first, and then click Deploy.

Answer

The configuration process for the Ethernet1/3 interface of the leaf02 switch will be very similar. Make sure that the Access Vlan value is the same ID value as you specified for your AI network VLAN ID. The description is now Link to PC-01, because it is the interface where PC-01 is connected.

After you click the Save button, you will get a notification in the bottom-right corner that you successfully edited the interfaces. Also, notice that the Deploy button is now clickable.

After clicking the Deploy button, you will be forwarded to the Deploy interfaces configuration screen showing you the Pending Configuration for the Ethernet1/3 interfaces.

Step 83

Click the Pending Config link for one of the interfaces to inspect the configuration.

Answer

Notice that NDFC is enabling PFC and QoS classification of the traffic on the Ethernet1/3 interface. This way, RoCEv2 traffic is immediately classified when it enters the fabric and the access interface will be able to use the PFC mechanism to resolve congestion if required.

Step 84

Click the X icon in the top-right corner to close the Pending Config window.

Answer

Step 85

In the bottom-right corner, click the Deploy Config button to deploy the configuration.

Answer

After a few seconds, you should see the Deploy interfaces configuration window, showing the progress of the deployment.

After deployment is done, NDFC will automatically return you the Edit Interfaces window. In the bottom-right corner, you should notice a notification about successful configuration deployment.

Step 86

Click the X button in the top-right corner to exit the Edit Interfaces window.

Answer

Notice that the interfaces now show Admin Status value UP. However, the Operational Status is still Down because you still didn't attach interfaces to the network.

Note

If the interface page remains the same for more than a couple of seconds, reload the page to refresh the view.

Step 87

Choose again both Ethernet1/3 interfaces on leaf switches, click the white Actions button, and then choose Multi-Attach from the drop-down list.

Answer

You should land on the Interface Multi-Attach window, where you can attach interfaces to the available networks. You should only see AI_Network_30000, which you configured for AI workloads.

Step 88

Choose AI_Network_30000 as the network where you will attach the selected Ethernet1/3 interfaces and click Next in the bottom-right corner.

Answer

Cisco NDFC will forward you to the Summary step where you can verify interface association.

Step 89

Make sure that the Proceed to Full Switch Deploy option is chosen, and click the Save button in the bottom-right corner.

Answer

After clicking the Save button, you should see the Deploy Configuration window. Both leaf switches have pending configuration.

Step 90

Click the Deploy All button to confirm deployment.

Answer

The deployment of this configuration takes a few seconds. When it is done, you should see the Deployment Completed status.

Step 91

Click the X button in the top-right corner to close the window.

Answer

Cisco NDFC will return you to the Interfaces tab. In this tab, you should notice that both Ethernet1/3 interfaces are now reporting Admin and Operational Status as UP.

Step 92

Click the terminal icon in the bottom toolbar, type the ping 192.168.1.254 command and press Enter to verify if Student VM can ping the gateway.

Answer

Notice that now the ping command works and that Student VM can reach the gateway.

Step 93

Use the ping 192.168.1.103 command to verify that Student VM can also reach PC-01, which is attached to the Ethernet1/3 interface on the leaf02 switch.

Answer

Notice that Student VM can also ping PC-01, which is connected to the other leaf switch.

With this task, you verified the connectivity between PC-01 and Student VM, which are communicating over a lossless network.

Note

The lab environment is using Cisco Nexus 9000v virtual switches. These switches allow you to configure QoS and congestion mechanisms for the AI network just like you would on the physical Cisco Nexus 9000 switches. Nonetheless, virtual Cisco Nexus 9000v switches cannot implement QoS policies and apply congestion mechanisms when congestion happens, that is only possible on physical Cisco Nexus switches. Therefore, this lab exercise does not cover congestion simulation.

Keep going!